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ABSTRACT 

When researchers evaluate social skills programs , 
they essentially are measuring progress towards a goal. How they 
measure this progress hinges on the precision of the goal statements, 
the identification of valid and realistic process and outcome 
indicators, and the selection of appropriate data and methods for 
collection. This paper outlines the necessary steps for evaluating 
social skills programs. It is argued that evaluating such programs 
involves the same processes used in any evaluation exercise. The 
paper opens with a discussion of how to establish aims and goals, 
including a seven-step process for setting goals. Process and outcome 
indicators are outlined next, along with a series of questions and 
specific indicators for social goals. The paper then discusses some 
of the data needed to measure progress and the instruments and 
procedures that can be used for measurement, such as interviews, 
questionnaires, sociometric techniques, and commercial tests. A 
discussion on commercial and other off-the-shelf instruments is also 
presented. It is emphasized that measuring progress requires careful 
planning but such efforts will be rewarded with increased confidence 
in evaluation conclusions. (RJM) 
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Before embarking on a description and discussion of the procedures involved in evaluating 
social skills programs, we first need to address the question of why. Why bother evaluating 
social skills programs in schools? Surely, it is enough for us to trust our own judgments about 
whether our programs are working or not. Yes, the practitioner's personal evaluation of 
program effectiveness is important, but we need to know that we are using our resources - time, 
energy and any financial cost involved - and our students' time in the most effective way 
possible. Most cireas of the school curriculum are able to be evaluated through the examination 
system. This does not generally apply to the social skills curriculum. However, if we are going 
to take the social development of our students seriously, then we must apply the same 
standards of program evaluation to this part of the curriculum as we do to mathematics, 

English and the other subjects that we teach in our schools. It is interesting to note that of some 
400 schools that proposed social skills programs for inclusion in the AGCA publication on 
Teaching ProSocial Skills to Adolescents (Prescott, 1995), less than a dozen were able to report 
that they had conducted any kind of formal evaluation into the effectiveness of their 
programs. 

We need to know to whether our programs to reduce bullying cire working Are the social 
problem solving programs that we are using resulting in increased social competence for our 
students? How effective are the conflict resolution strategies that we are teaching our 
students? Are we making any headway in helping that 10% of our students population who do 
not have any friends at all at school? If so, what is working, what is making the difference? 

Evaluating social skills programs involves essentially the same processes that would be 
employed in any evaluation exercise: (1) clear statements of the aims and goals of the program; 
(2) descriptions of process- and/or outcome-indicators - that is, descriptions of the behaviour 
and feelings of students, teachers, parents and others that we would expect to indicate that we 
are making progress towards our goals; (3) identification of the data, or information that we 
would need to let us know if we are achieving (or have achieved) our goals; (4) identification 
of the methods and instruments that we could use to collect these data; and (5) procedures for 
analysing and interpreting this information in light of the aims and goals of our programs. 

AIMS AND GOALS 

The first question to ask is: what is the overall aim of my project? The purpose of such an aim 
would be to declare one's intent to do something - in this case about the social world of school - 
and to describe a general outcome that is desired. An example of such an aim would be: The aim 
of this project is to reduce the amount of playground violence in the school. 

Evaluation of broadly stated aims can often be difficult. However, if these aims are 
operationalised into specific goals, then evaluation becomes more feasible. Gerard Egan (1994) 
suggests seven requirements for setting goals. A goal should be: 

• stated as an accomplishment 

• behaviourally clear and specific 

• measurable or verifiable 

• substantive 

• in keeping with the values of those concerned, and 

• set within a reasonable time frame 
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Goals stated in such a way are more likely to be achieved, than those that are left vague and 
woolly, such as: "our goal is the provide a safe and happy environment for all students". This 
is indeed a laudable goal, but it might be difficult to measure the school's progress towards its 
accomplishment. However, if we were to ask questions that operationalised this goal, such as: 
xvhat sort of things would students be doing in a peaceful school? or future-oriented questions 
such as: what would be happening in our peaceful school of the future that is not happening 
now? , we would be in a better position to know the extent to which our efforts were helping us 
make progress towards our goal. 



PROCESS AND OUTCOME INDICATORS 

We can again turn to Egan (1994) for further examples of the kinds of future-oriented questions 
that will help us know when we have achieved our goals, or the extent to which we are making 
progress towards their achievement. Egan suggests that we pose questions like: 

• What would the current situation look like if it were better? 

• What would be happening if things were fractionally better? 

• What would be happening if things were dramatically better? 

• What would students (or staff) be doing that they are not doing now/ 

• What would they stop doing? 

• What accomplishments would be in place? 

• What would be happening that is not happening now? 

• What would this opportunity look like if it were developed? 

In reflecting upon our own programs, we need to ask ourselves: what behaviour or action would 
indicate that are goals are being achieved? Questions like this, posed in the present tense, 
lead us to process indicators which contribute to formative evaluation procedures. The kinds of 
questions that ask what sorts of actions indicate that are goal have been achieved (past tense) 
lead us to outcome indicators which serve summative evaluation procedures. Examples of 
process and outcome indicators for social goals could include: 

• fewer outbursts in class or playground 

• students are more cooperative 

• students are less isolated 

• fewer students are spending time in "time out" 

• a particular student is smiling more 

• teacher reporting improved relationships with his/her students 

• parent reporting less difficulty in getting student off to school 

• students seeking less help from teachers to resolve conflicts during recess time 

Process and outcome indicators can be identified to sample students' and teachers' feelings and 
behaviour for both groups are likely to be affected as a result of a social skills program. Such 
indicators may extend to descriptions of new whole-of-school procedures, and also include 
changes in parents' actions and feelings. 



DATA THAT ARE NEEDED 

While process and outcome indicators can give us an idea of the kinds of changes to look for in 
the evaluation process, we may need to measure certain behaviours in order to make a valid 
judgments about our goals. Basically, we ask: what information will let us know if we are 
achieving (or have achieved) our goals? Applied to some of the indicators above, we may need 
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to ask: what information will tell us if students are less isolated, or more cooperative, or 
spending less time in "time out"? Similarly, we may need to identify the kinds of information 
that will let us know if teachers are experiencing improved relationships with their students, 
or if students are being harassed less by their classmates. This kind of information, usually 
quantitative in nature, may be essential in determining whether on not goals have been 
achieved. 

A second type of data may allow us to go beyond merely answering the question: "to what extent 
have goals between achieved?" Here, we ask the question: "what information will be useful to 
us in understanding what is happening?" For example, a recent study by Owens and MacMullin 
(1995) reported the incidence of different types of aggression among boys and girls at various 
ages. This study found more indirect aggression amongst adolescent girls than amongst 
adolescent boys. These same procedures could be used to measure any changes in aggression rates 
that may result from an intervention program. However, the data that Owens & MacMullin 
collected tell us very little about the nature of indirect aggression among adolescent girls. 
School-based evaluation projects may focus on what is happening and why it is happening. We 
need to go beyond the quantitative appraisals of our social skills programs to ask questions like: 
why was this program successful, or not successful? What part of our intervention made the real 
difference? What kinds of things need to happen in order for our programs to be more effective? 
These kinds of questions suggest qualitative as well as quantitative data. 



INSTRUMENTS AND PROCEDURES 

Having determined the kinds of information that are needed in order to answer our evaluations 
questions, we now need to select sources of that information and procedures and instruments to 
gather it. Data sources and gathering techniques include: 

interviews 

questionnaires 

checklists 

records of events 

journals or diaries 

samples of students' work 

direct observation notes 

photographs 

video- and audiotapes 

personal reflections (field notes) 

teacher made tests 

sociometric techniques 

commercial tests 

A number of things need to be considered when we develop our own instruments. When 
employing observations, we need to decide whether to use naturalist and/or contrived 
situations. We need to consider whether we will be covert, overt, or participant observers. We 
also need to decide upon direct or video-recorded methods of observing, and whether our 
observations will be scheduled in any way or totally open-ended. 

Similarly, our plans to interview people need to consider whether to use schedules or open- 
ended questions. Of course, much thought needs to be given to the questions themselves. We will 
also need to give thought to the methods we will employ in collecting the interview data. 

These are likely to include: tape-recording (and later transcribing) the whole interview, 
scribing answers as they are delivered, or attending fully to the interviewee and writing up the 



responses from memory post hoc. Decisions about individual versus group interviews may also 
be important. 

In developing and using surveys and questionnaires we will need to attend to questions of 
sampling - both the domain that we wish to investigate (validity) as well as the population of 
people from whom we seek information (reliability). Other issues involved include decisions 
about face-to-face versus mail versus telephone surveying, types of questions and types of 
answering systems. 

While home-made procedures and instruments provide us with an opportunity to collect 
information that is specific to our evaluate tasks, we need to know that these endeavours may 
be particularly time-consuming, and sometimes may involve us in re-inventing the wheel. 



COMMERCIAL AND OTHER OFF-THE-SHELF INSTRUMENTS 

There are a number of commercial instruments available in Australia to measure aspects of 
children's social competence. Domains include: self esteem, social status and peer relationships, 
social skills, adaptive behaviour, behaviour checklists, measures of classroom environment, 
and inventories of problematic situations. For a complete list of these instruments see 
MacMullin (1995). 

Issues to bear in mind in using off-the-shelf instruments include: permission, suitability, time, 
expertise, and cost. It is a generally excepted practice that unless the instrument or procedure 
that you use can be considered part of normal teaching, parental permission is required before 
children can be subjected to your procedures. With respect to suitability, many off-the-shelf 
instruments are designed for individual administration and may not be suitable for use with 
groups of students. We should also be aware that some instruments are designed for particular 
groups of students, e.g. those with an intellectual disability, or those with a behaviour 
disorder. These may not be suitable for general administration. 

It is useful to point out that many instruments are particularly time-consuming, and some may 
make excessive demands on teachers and/or students. Financial costs will also need to be 
considered as most test protocols are covered by copyright. 

A final word of advice concerns expertise. While many of the commercial instruments are able 
to be administered by teachers, some are restricted to psychologists only. Look out for any legal 
restrictions that may be placed on particular instruments and recognise the skills that an 
educational psychologist may be able to bring to your evaluation project. 



CONCLUSION 

When we attempt to evaluate social skills programs, we are essentially asking: to what extent 
have we made progress towards the goals of our program? Our capacity to cinswer this question 
hinges largely on the precision of our goal statements, our identification of valid and realistic 
process and outcome indicators, and our selection of appropriate data and methods for their 
collection. This process requires careful planning. However, the time and care devoted to such 
planning is not likely to be wasted, for evaluators should be rewarded with a greater sense of 
confidence in the conclusions that they are able to draw from their work. 
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